{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Local Automated Machine Learning Model with ACI Deployment for Predicting Sentence Similarity"
   ]
  },
    {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/nlp/examples/sentence_similarity/automl_local_deployment_aci.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to use [Azure Machine Learning Service's](https://azure.microsoft.com/en-us/services/machine-learning-service/\n",
    ") Automated Machine Learning ([AutoML](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-automated-ml\n",
    ")) locally to automate machine learning model selection and tuning and how to use Azure Container Instance ([ACI](https://azure.microsoft.com/en-us/services/container-instances/\n",
    ")) for deployment. We utilize the STS Benchmark dataset to predict sentence similarity and utilize AutoML's text preprocessing features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. [Introduction](#1.-Introduction)  \n",
    "     [1.1 What is Azure AutoML?](#1.1-What-is-Azure-AutoML?)  \n",
    "     [1.2 Modeling Problem](#1.2-Modeling-Problem)  \n",
    "1. [Data Preparation](#2.-Data-Preparation)  \n",
    "1. [Create AutoML Run](#3.-Create-AutoML-Run)    \n",
    "    [3.1 Link to or create a Workspace](#3.1-Link-to-or-create-a-Workspace)  \n",
    "    [3.2 Create AutoMLConfig object](#3.2-Create-AutoMLConfig-object)\n",
    "    [3.3 Run Experiment](#3.3-Run-Experiment)      \n",
    "1. [Deploy Sentence Similarity Model](#4.-Deploy-Sentence-Similarity-Model)  \n",
    "    [4.1 Retrieve the Best Model](#4.1-Retrieve-the-Best-Model)  \n",
    "    [4.2 Register the Fitted Model for Deployment](#4.2-Register-the-Fitted-Model-for-Deployment)   \n",
    "    [4.3 Create an Entry Script](#4.3-Create-an-Entry-Script)   \n",
    "    [4.4 Create a YAML File for the Environment](#4.4-Create-a-YAML-File-for-the-Environment)  \n",
    "    [4.5 Create a Container Image](#4.5-Create-a-Container-Image)    \n",
    "    [4.6 Deploy the Image as a Web Service to Azure Container Instance](#4.6-Deploy-the-Image-as-a-Web-Service-to-Azure-Container-Instance)  \n",
    "    [4.7 Test Deployed Model](#4.7-Test-Deployed-Model)     \n",
    "1. [Clean](#5-Clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 What is Azure AutoML?\n",
    "\n",
    "Automated machine learning ([AutoML](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-automated-ml)) is a capability of Microsoft's [Azure Machine Learning service](https://azure.microsoft.com/en-us/services/machine-learning-service/\n",
    "). The goal of AutoML is to improve the productivity of data scientists and democratize AI by allowing for the rapid development and deployment of machine learning models. To acheive this goal, AutoML automates the process of selecting a ML model and tuning the model. All the user is required to provide is a dataset (suitable for a classification, regression, or time-series forecasting problem) and a metric to optimize in choosing the model and hyperparameters. The user is also given the ability to set time and cost constraints for the model selection and tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://nlpbp.blob.core.windows.net/images/automl.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The AutoML model selection and tuning process can be easily tracked through the Azure portal or directly in python notebooks through the use of widgets. AutoML quickly selects a high quilty machine learning model tailored for your prediction problem. In this notebook, we walk through the steps of preparing data, setting up an AutoML experiment, and evaluating the results of our best model. More information about running AutoML experiments in Python can be found [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Modeling Problem\n",
    "\n",
    "The regression problem we will demonstrate is predicting sentence similarity scores on the STS Benchmark dataset. The [STS Benchmark dataset](http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark#STS_benchmark_dataset_and_companion_dataset) contains a selection of English datasets that were used in Semantic Textual Similarity (STS) tasks 2012-2017. The dataset contains 8,628 sentence pairs with a human-labeled integer representing the sentences' similarity (ranging from 0, for no meaning overlap, to 5, meaning equivalence). The sentence pairs will be embedded using AutoML's built-in preprocessing, so we'll pass the sentences directly into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Turning diagnostics collection on. \n",
      "System version: 3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)]\n",
      "Azure ML SDK Version: 1.0.48\n",
      "Pandas version: 0.23.4\n"
     ]
    }
   ],
   "source": [
    "# Set the environment path to find NLP\n",
    "import sys\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "import time\n",
    "import os\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import numpy as np\n",
    "import torch\n",
    "import sys\n",
    "from scipy.stats import pearsonr\n",
    "from scipy.spatial import distance\n",
    "from sklearn.externals import joblib\n",
    "import json\n",
    "import scrapbook as sb\n",
    "\n",
    "# Import utils\n",
    "from utils_nlp.azureml import azureml_utils\n",
    "from utils_nlp.dataset import stsbenchmark\n",
    "from utils_nlp.dataset.preprocess import (\n",
    "    to_lowercase,\n",
    "    to_spacy_tokens,\n",
    "    rm_spacy_stopwords,\n",
    ")\n",
    "from utils_nlp.common.timer import Timer\n",
    "\n",
    "# Tensorflow dependencies for Google Universal Sentence Encoder\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "\n",
    "# AzureML packages\n",
    "import azureml as aml\n",
    "import logging\n",
    "from azureml.telemetry import set_diagnostics_collection\n",
    "\n",
    "set_diagnostics_collection(send_diagnostics=True)\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "from azureml.train.automl.run import AutoMLRun\n",
    "from azureml.core.webservice import AciWebservice, Webservice\n",
    "from azureml.core.image import ContainerImage\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Azure ML SDK Version:\", aml.core.VERSION)\n",
    "print(\"Pandas version: {}\".format(pd.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "BASE_DATA_PATH = \"../../data\"\n",
    "CPU_CORES = 1\n",
    "MEMORY_GB = 8\n",
    "\n",
    "# Define the settings for AutoML\n",
    "automl_task = \"regression\"\n",
    "automl_iteration_timeout = 15\n",
    "automl_iterations = 50\n",
    "automl_metric = \"spearman_correlation\"\n",
    "automl_preprocess = True\n",
    "automl_model_blacklist = ['XGBoostRegressor']\n",
    "\n",
    "config_path = (\n",
    "    \"./.azureml\"\n",
    ")  # Path to the directory containing config.json with azureml credentials\n",
    "\n",
    "webservice_name = \"aci-automl-service\" #name for webservice; must be unique within your workspace\n",
    "\n",
    "# Azure resources\n",
    "subscription_id = \"YOUR_SUBSCRIPTION_ID\"\n",
    "resource_group = \"YOUR_RESOURCE_GROUP_NAME\"  \n",
    "workspace_name = \"YOUR_WORKSPACE_NAME\"  \n",
    "workspace_region = \"YOUR_WORKSPACE_REGION\" #Possible values eastus, eastus2 and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"task\": automl_task,  # type of task: classification, regression or forecasting\n",
    "    \"debug_log\": \"automated_ml_errors.log\",\n",
    "    \"path\": \"./automated-ml-regression\",\n",
    "    \"iteration_timeout_minutes\": automl_iteration_timeout,  # How long each iteration can take before moving on\n",
    "    \"iterations\": automl_iterations,  # Number of algorithm options to try\n",
    "    \"primary_metric\": automl_metric,  # Metric to optimize\n",
    "    \"preprocess\": automl_preprocess,  # Whether dataset preprocessing should be applied\n",
    "    \"blacklist_models\": automl_model_blacklist #exclude this model due to installation issues\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STS Benchmark Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As described above, the STS Benchmark dataset contains 8.6K sentence pairs along with a human-annotated score for how similiar the two sentences are. We will load the training, development (validation), and test sets provided by STS Benchmark and preprocess the data (lowercase the text, drop irrelevant columns, and rename the remaining columns) using the utils contained in this repo. Each dataset will ultimately have three columns: _sentence1_ and _sentence2_ which contain the text of the sentences in the sentence pair, and _score_ which contains the human-annotated similarity score of the sentence pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 401/401 [00:02<00:00, 146KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data downloaded to ../../data\\raw\\stsbenchmark\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 401/401 [00:02<00:00, 172KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data downloaded to ../../data\\raw\\stsbenchmark\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████| 401/401 [00:02<00:00, 182KB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data downloaded to ../../data\\raw\\stsbenchmark\n"
     ]
    }
   ],
   "source": [
    "# Load in the raw datasets as pandas dataframes\n",
    "train_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split=\"train\")\n",
    "dev_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split=\"dev\")\n",
    "test_raw = stsbenchmark.load_pandas_df(BASE_DATA_PATH, file_split=\"test\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean each dataset by lowercasing text, removing irrelevant columns,\n",
    "# and renaming the remaining columns\n",
    "train_clean = stsbenchmark.clean_sts(train_raw)\n",
    "dev_clean = stsbenchmark.clean_sts(dev_raw)\n",
    "test_clean = stsbenchmark.clean_sts(test_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert all text to lowercase\n",
    "train = to_lowercase(train_clean)\n",
    "dev = to_lowercase(dev_clean)\n",
    "test = to_lowercase(test_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set has 5749 sentences\n",
      "Development set has 1500 sentences\n",
      "Testing set has 1379 sentences\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set has {} sentences\".format(len(train)))\n",
    "print(\"Development set has {} sentences\".format(len(dev)))\n",
    "print(\"Testing set has {} sentences\".format(len(test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>score</th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.00</td>\n",
       "      <td>a plane is taking off.</td>\n",
       "      <td>an air plane is taking off.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.80</td>\n",
       "      <td>a man is playing a large flute.</td>\n",
       "      <td>a man is playing a flute.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.80</td>\n",
       "      <td>a man is spreading shreded cheese on a pizza.</td>\n",
       "      <td>a man is spreading shredded cheese on an uncoo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.60</td>\n",
       "      <td>three men are playing chess.</td>\n",
       "      <td>two men are playing chess.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.25</td>\n",
       "      <td>a man is playing the cello.</td>\n",
       "      <td>a man seated is playing the cello.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   score                                      sentence1  \\\n",
       "0   5.00                         a plane is taking off.   \n",
       "1   3.80                a man is playing a large flute.   \n",
       "2   3.80  a man is spreading shreded cheese on a pizza.   \n",
       "3   2.60                   three men are playing chess.   \n",
       "4   4.25                    a man is playing the cello.   \n",
       "\n",
       "                                           sentence2  \n",
       "0                        an air plane is taking off.  \n",
       "1                          a man is playing a flute.  \n",
       "2  a man is spreading shredded cheese on an uncoo...  \n",
       "3                         two men are playing chess.  \n",
       "4                 a man seated is playing the cello.  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Create AutoML Run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AutoML can be used for classification, regression or timeseries experiments. Each experiment type has corresponding machine learning models and metrics that can be optimized (see [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train)) and the options will be delineated below. As a first step we connect to an existing workspace or create one if it doesn't exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Link to or create a Workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell looks to set up the connection to your [Azure Machine Learning service Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace). You can choose to connect to an existing workspace or create a new one. \n",
    "\n",
    "**To access an existing workspace:**\n",
    "1. If you have a `config.json` file, you do not need to provide the workspace information; you will only need to update the `config_path` variable that is defined above which contains the file.\n",
    "2. Otherwise, you will need to supply the following:\n",
    "    * The name of your workspace\n",
    "    * Your subscription id\n",
    "    * The resource group name\n",
    "\n",
    "**To create a new workspace:**\n",
    "\n",
    "Set the following information:\n",
    "* A name for your workspace\n",
    "* Your subscription id\n",
    "* The resource group name\n",
    "* [Azure region](https://azure.microsoft.com/en-us/global-infrastructure/regions/) to create the workspace in, such as `eastus2`. \n",
    "\n",
    "This will automatically create a new resource group for you in the region provided if a resource group with the name given does not already exist. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing interactive authentication. Please follow the instructions on the terminal.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - Note, we have launched a browser for you to login. For old experience with device code, use \"az login --use-device-code\"\n",
      "WARNING - You have logged in. Now let us find all the subscriptions to which you have access...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Interactive authentication successfully completed.\n"
     ]
    }
   ],
   "source": [
    "ws = azureml_utils.get_or_create_workspace(\n",
    "    config_path=config_path,\n",
    "    subscription_id=subscription_id,\n",
    "    resource_group=resource_group,\n",
    "    workspace_name=workspace_name,\n",
    "    workspace_region=workspace_region,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    \"Workspace name: \" + ws.name,\n",
    "    \"Azure region: \" + ws.location,\n",
    "    \"Subscription id: \" + ws.subscription_id,\n",
    "    \"Resource group: \" + ws.resource_group,\n",
    "    sep=\"\\n\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Create AutoMLConfig object\n",
    "Next, we specify the parameters for the AutoMLConfig class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**task**  \n",
    "AutoML supports the following base learners for the regression task: Elastic Net, Light GBM, Gradient Boosting, Decision Tree, K-nearest Neighbors, LARS Lasso, Stochastic Gradient Descent, Random Forest, Extremely Randomized Trees, XGBoost, DNN Regressor, Linear Regression. In addition, AutoML also supports two kinds of ensemble methods: voting (weighted average of the output of multiple base learners) and stacking (training a second \"metalearner\" which uses the base algorithms' predictions to predict the target variable). Specific base learners can be included or excluded in the parameters for the AutoMLConfig class (whitelist_models and blacklist_models) and the voting/stacking ensemble options can be specified as well (enable_voting_ensemble and enable_stack_ensemble)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**preprocess**  \n",
    "AutoML also has advanced preprocessing methods, eliminating the need for users to perform this manually. Data is automatically scaled and normalized but an additional parameter in the AutoMLConfig class enables the use of more advanced techniques including imputation, generating additional features, transformations, word embeddings, etc. (full list found [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-create-portal-experiments#preprocess)). Note that algorithm-specific preprocessing will be applied even if preprocess=False. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**primary_metric**  \n",
    "The regression metrics available are the following: Spearman Correlation (spearman_correlation), Normalized RMSE (normalized_root_mean_squared_error), Normalized MAE (normalized_mean_absolute_error), and R2 score (r2_score) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Constraints:**  \n",
    "There is a cost_mode parameter to set cost prediction modes (see options [here](https://docs.microsoft.com/en-us/python/api/azureml-train-automl/azureml.train.automl.automlconfig?view=azure-ml-py)). To set constraints on time there are multiple parameters including experiment_exit_score (target score to exit the experiment after achieving), experiment_timeout_minutes (maximum amount of time for all combined iterations), and iterations (total number of different algorithm and parameter combinations to try)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: we are directly passing in sentence pairs as data because we are relying upon AutoML's built-in preprocessing (by setting preprocess = True in the AutoMLConfig parameters) to perform the embedding step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train.drop(\"score\", axis=1).values\n",
    "y_train = train[\"score\"].values.flatten()\n",
    "X_validation = dev.drop(\"score\", axis=1).values\n",
    "y_validation = dev[\"score\"].values.flatten()\n",
    "\n",
    "# local compute\n",
    "automated_ml_config = AutoMLConfig(\n",
    "    X=X_train,\n",
    "    y=y_train,\n",
    "    X_valid=X_validation,\n",
    "    y_valid=y_validation,\n",
    "    verbosity=logging.ERROR,\n",
    "    **automl_settings  # where the autoML main settings are defined\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Run Experiment\n",
    "\n",
    "Run the experiment locally and inspect the results using a widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = Experiment(ws, \"NLP-SS-automl\")\n",
    "local_run = experiment.submit(automated_ml_config, show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#local_run.cancel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of the completed run can be visualized in two ways. First, by using a RunDetails widget as shown in the cell below. Second, by accessing the [Azure portal](https://portal.azure.com), selecting your workspace, clicking on _Experiments_ and then selecting the name and run number of the experiment you want to inspect. Both these methods will show the results and duration for each iteration (algorithm tried), a visualization of the results, and information about the run including the compute target, primary metric, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the run details using the provided widget\n",
    "RunDetails(local_run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://nlpbp.blob.core.windows.net/images/autoMLwidget.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Deploy Sentence Similarity Model\n",
    "Deploying an Azure Machine Learning model as a web service creates a REST API. You can send data to this API and receive the prediction returned by the model.\n",
    "In general, you create a webservice by deploying a model as an image to a Compute Target.\n",
    "\n",
    "Some of the Compute Targets are: \n",
    "1. Azure Container Instance\n",
    "2. Azure Kubernetes Service\n",
    "3. Local web service\n",
    "\n",
    "The general workflow for deploying a model is as follows:\n",
    "1. Register a model\n",
    "2. Prepare to deploy\n",
    "3. Deploy the model to the compute target\n",
    "4. Test the deployed model (webservice)\n",
    "\n",
    "In this notebook, we walk you through the process of creating a webservice running on Azure Container Instance by deploying an AutoML model as an image. ACI is typically used for low scale, CPU-based workloads. (You can find more information on deploying and serving models [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where))\n",
    "\n",
    "## 4.1 Retrieve the Best Model\n",
    "Now we can identify the model that maximized performance on a given metric (spearman correlation in our case) using the `get_output` method which returns the best_run (AutoMLRun object with information about the experiment) and fitted_model ([Pipeline]((https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-getting-started.ipynb)) object) across all iterations. Overloads on `get_output` allow you to retrieve the best run and fitted model for any logged metric or for a particular iteration. \n",
    "\n",
    "The different steps that make up the pipeline can be accessed through `fitted_model.named_steps` and information about data preprocessing is available through `fitted_model.named_steps['datatransformer'].get_featurization_summary()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = local_run.get_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Register the Fitted Model for Deployment\n",
    "\n",
    "Registering a model means registering one or more files that make up a model. The Machine Learning models are registered in your current Aure Machine Learning Workspace. The model can either come from Azure Machine Learning or another location, such as your local machine. \n",
    "Below we show how a model is registered from the results of an experiment run. If neither metric nor iteration are specified in the register_model call, the iteration with the best primary metric is registered.\n",
    "\n",
    "See other ways to register a model [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registering model AutoML29e4bb98ebest\n",
      "AutoML29e4bb98ebest\n"
     ]
    }
   ],
   "source": [
    "description = \"AutoML Model\"\n",
    "tags = {\"area\": \"nlp\", \"type\": \"sentence similarity automl\"}\n",
    "name = \"automl\"\n",
    "model = local_run.register_model(description=description, tags=tags)\n",
    "\n",
    "print(local_run.model_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 Create an Entry Script\n",
    "In this section we show an example of an entry script, which is called from the deployed webservice. `score.py` is our entry script. The script must contain:\n",
    "1. init() - This function loads the model in a global object.\n",
    "2. run() - This function is used for model prediction. The inputs and outputs to `run()` typically use JSON for serialization and deserilization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting score.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile score.py\n",
    "import pickle\n",
    "import json\n",
    "import numpy\n",
    "import azureml.train.automl\n",
    "from sklearn.externals import joblib\n",
    "from azureml.core.model import Model\n",
    "\n",
    "\n",
    "def init():\n",
    "    global model\n",
    "    model_path = Model.get_model_path(\n",
    "        model_name=\"<<modelid>>\"\n",
    "    )  # this name is model.id of model that we want to deploy\n",
    "    # deserialize the model file back into a sklearn model\n",
    "    model = joblib.load(model_path)\n",
    "\n",
    "\n",
    "def run(rawdata):\n",
    "    try:\n",
    "        data = json.loads(rawdata)[\"data\"]\n",
    "        data = numpy.array(data)\n",
    "        result = model.predict(data)\n",
    "    except Exception as e:\n",
    "        result = str(e)\n",
    "        return json.dumps({\"error\": result})\n",
    "    return json.dumps({\"result\": result.tolist()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Substitute the actual model id in the script file.\n",
    "script_file_name = \"score.py\"\n",
    "\n",
    "with open(script_file_name, \"r\") as cefr:\n",
    "    content = cefr.read()\n",
    "\n",
    "with open(script_file_name, \"w\") as cefw:\n",
    "    cefw.write(content.replace(\"<<modelid>>\", local_run.model_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 Create a YAML File for the Environment\n",
    "\n",
    "To ensure the fit results are consistent with the training results, the SDK dependency versions need to be the same as the environment that trains the model. The following cells create a file, autoenv.yml, which specifies the dependencies from the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_run = AutoMLRun(experiment=experiment, run_id=local_run.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_iteration = int(\n",
    "    best_run.id.split(\"_\")[-1]\n",
    ")  # get the appended iteration number for the best model\n",
    "dependencies = ml_run.get_run_sdk_dependencies(iteration=best_iteration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'azureml-widgets': '1.0.48',\n",
       " 'azureml-train': '1.0.48',\n",
       " 'azureml-train-restclients-hyperdrive': '1.0.48',\n",
       " 'azureml-train-core': '1.0.48',\n",
       " 'azureml-train-automl': '1.0.48',\n",
       " 'azureml-telemetry': '1.0.48',\n",
       " 'azureml-sdk': '1.0.48',\n",
       " 'azureml-pipeline': '1.0.48',\n",
       " 'azureml-pipeline-steps': '1.0.48',\n",
       " 'azureml-pipeline-core': '1.0.48',\n",
       " 'azureml-mlflow': '1.0.48',\n",
       " 'azureml-dataprep': '1.1.8',\n",
       " 'azureml-dataprep-native': '13.0.0',\n",
       " 'azureml-core': '1.0.48',\n",
       " 'azureml-automl-core': '1.0.48'}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add dependencies in the yaml file from the above cell. You must specify the version of \"azureml-sdk[automl]\" while creating the yaml file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'automlenv.yml'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myenv = CondaDependencies.create(\n",
    "    conda_packages=[\"numpy\", \"scikit-learn==0.21.2\", \"py-xgboost<=0.80\", \"pandas==0.24.2\"],\n",
    "    pip_packages=[\"azureml-sdk[automl]==1.0.48.*\"],\n",
    "    python_version=\"3.6.8\",\n",
    ")\n",
    "\n",
    "conda_env_file_name = \"automlenv.yml\"\n",
    "myenv.save_to_file(\".\", conda_env_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5 Create a Container Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we create a container image which is wrapper containing the entry script, yaml file with package dependencies and the model. The created image is then deployed as a webservice in the next step. This step can take up to 10 minutes and even longer if the model is large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating image\n",
      "Running...................................................\n",
      "Succeeded\n",
      "Image creation operation finished for image automl-image:20, operation \"Succeeded\"\n"
     ]
    }
   ],
   "source": [
    "image_config = ContainerImage.image_configuration(\n",
    "    execution_script=script_file_name,\n",
    "    runtime=\"python\",\n",
    "    conda_file=conda_env_file_name,\n",
    "    description=\"Image with automl model\",\n",
    "    tags={\"area\": \"nlp\", \"type\": \"sentencesimilarity automl\"},\n",
    ")\n",
    "\n",
    "image = ContainerImage.create(\n",
    "    name=\"automl-image\",\n",
    "    # this is the model object\n",
    "    models=[model],\n",
    "    image_config=image_config,\n",
    "    workspace=ws,\n",
    ")\n",
    "\n",
    "image.wait_for_creation(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the above step fails, then use the below command to see logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(image.image_build_log_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6 Deploy the Image as a Web Service to Azure Container Instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Azure Container Instances are mostly used for deploying your models as a web service if one or more of the following conditions are true:\n",
    "1. You need to quickly deploy and validate your model.\n",
    "2. You are testing a model that is under development.\n",
    "\n",
    "To set them up properly, we need to indicate the number of CPU cores and the amount of memory we want to allocate to our web service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the web service configuration\n",
    "aci_config = AciWebservice.deploy_configuration(\n",
    "    cpu_cores=CPU_CORES, memory_gb=MEMORY_GB\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step to deploying our web service is to call `WebService.deploy_from_image()`. This function uses the Docker image and the deployment configuration we created above to perform the following:\n",
    "1. Deploy the docker image to an Azure Container Instance\n",
    "2. Call the init() function in our scoring file\n",
    "3. Provide an HTTP endpoint for scoring calls\n",
    "\n",
    "The deploy_from_image method requires the following parameters:\n",
    "\n",
    "1. workspace: the workspace containing the service\n",
    "2. name: a unique name used to identify the service in the workspace\n",
    "3. image: a docker image object that contains the environment needed for scoring/inference\n",
    "4. deployment_config: a configuration object describing the compute type\n",
    "\n",
    "**Note:** The web service creation can take a few minutes "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating service\n",
      "Running............................\n",
      "SucceededACI service creation operation finished, operation \"Succeeded\"\n",
      "Healthy\n"
     ]
    }
   ],
   "source": [
    "# deploy image as web service\n",
    "aci_service = Webservice.deploy_from_image(\n",
    "    workspace=ws, name=webservice_name, image=image, deployment_config=aci_config\n",
    ")\n",
    "\n",
    "aci_service.wait_for_deployment(show_output=True)\n",
    "print(aci_service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fetch logs to debug in case of failures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(aci_service.get_logs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to reuse an existing service versus creating a new one, call the webservice with the name. You can look up all the deployed webservices under deployment in the Azure Portal. Below is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aci_service = Webservice(workspace=ws, name='<<serive-name>>')\n",
    "\n",
    "# to use the webservice\n",
    "# aci_service.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.7 Test Deployed Model\n",
    "\n",
    "Testing the deployed model means running the created webservice. <br>\n",
    "The deployed model can be tested by passing a list of sentence pairs. The output will be a score between 0 and 5, with 0 indicating no meaning overlap between the sentences and 5 meaning equivalence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [\n",
    "    [\"This is sentence1\", \"This is sentence1\"],\n",
    "    [\"A hungry cat.\", \"A sleeping cat\"],\n",
    "    [\"Its summer time \", \"Winter is coming\"],\n",
    "]\n",
    "data = {\"data\": sentences}\n",
    "data = json.dumps(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 0.3863\n",
      "Number of samples predicted: 3\n",
      "[1.8922695576076576, 2.981616317499634, 1.895254173398724]\n"
     ]
    }
   ],
   "source": [
    "# Set up a Timer to see how long the model takes to predict\n",
    "t = Timer()\n",
    "\n",
    "t.start()\n",
    "score = aci_service.run(input_data=data)\n",
    "t.stop()\n",
    "\n",
    "print(\"Time elapsed: {}\".format(t))\n",
    "\n",
    "result = json.loads(score)\n",
    "try:\n",
    "    output = result[\"result\"]\n",
    "    print(\"Number of samples predicted: {}\".format(len(output)))\n",
    "    print(output)\n",
    "except:\n",
    "    print(result[\"error\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll calculate the Pearson Correlation on the test set.\n",
    "\n",
    "**What is Pearson Correlation?**\n",
    "\n",
    "Our evaluation metric is Pearson correlation ($\\rho$) which is a measure of the linear correlation between two variables. The formula for calculating Pearson correlation is as follows:  \n",
    "\n",
    "$$\\rho_{X,Y} = \\frac{E[(X-\\mu_X)(Y-\\mu_Y)]}{\\sigma_X \\sigma_Y}$$\n",
    "\n",
    "This metric takes a value in [-1,1] where -1 represents a perfect negative correlation, 1 represents a perfect positive correlation, and 0 represents no correlation. We utilize the Pearson correlation metric as this is the main metric that [SentEval](http://nlpprogress.com/english/semantic_textual_similarity.html), a widely-used evaluation toolkit for evaluation sentence representations, uses for the STS Benchmark dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_y = test[\"score\"].values.flatten()\n",
    "test_x = test.drop(\"score\", axis=1).values.tolist()\n",
    "\n",
    "data = {\"data\": test_x}\n",
    "data = json.dumps(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time elapsed: 1.1874\n",
      "Number of samples predicted: 1379\n"
     ]
    }
   ],
   "source": [
    "# Set up a Timer to see how long the model takes to predict\n",
    "t = Timer()\n",
    "\n",
    "t.start()\n",
    "score = aci_service.run(input_data=data)\n",
    "t.stop()\n",
    "\n",
    "print(\"Time elapsed: {}\".format(t))\n",
    "\n",
    "result = json.loads(score)\n",
    "try:\n",
    "    output = result[\"result\"]\n",
    "    print(\"Number of samples predicted: {}\".format(len(output)))\n",
    "except:\n",
    "    print(result[\"error\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get Pearson Correlation\n",
    "pearson = pearsonr(output, test_y)[0]\n",
    "print(pearson)\n",
    "\n",
    "sb.glue(\"pearson_correlation\", pearson)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this notebook is to demonstrate how to use AutoML locally and then deploy the model to Azure Container Instance quickly. The model utilizes the built-in capabilities of AutoML to embed our sentences. The model performance on its own, without tweaking, is not very strong with this particular dataset. For a more advanced model, see [AutoML with Pipelines Deployment AKS](automl_with_pipelines_deployment_aks.ipynb) for much stronger performance on the same task. This notebook utilizes AzureML Pipelines to explicitly embed our sentences using the Google Universal Sentence Encoder (USE) model. For our dataset, the Google USE embeddings result in superior model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Clean up\n",
    "Throughout the notebook, we used a workspace and Azure container instances. To get a sense of the cost we incurred, we can refer to this [calculator](https://azure.microsoft.com/en-us/pricing/calculator/). We can also navigate to the [Cost Management + Billing](https://ms.portal.azure.com/#blade/Microsoft_Azure_Billing/ModernBillingMenuBlade/Overview) pane on the portal, click on our subscription ID, and click on the Cost Analysis tab to check our credit usage.\n",
    "<br><br>\n",
    "In order not to incur extra costs, let's delete the resources we no longer need.\n",
    "<br><br>\n",
    "Once we have verified that our web service works well on ACI, we can delete it. This helps reduce [costs](https://azure.microsoft.com/en-us/pricing/details/container-instances/), since the container group we were paying for no longer exists, and allows us to keep our workspace clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aci_service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, the main resource we are paying for is the Standard Azure Container Registry (ACR), which contains our Docker image. Details on pricing are available [here](https://azure.microsoft.com/en-us/pricing/details/container-registry/).\n",
    "\n",
    "We may decide to use our Docker image in a separate ACI or even in an AKS deployment. In that case, we should keep it available in our workspace. However, if we no longer have a use for it, we can delete it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# docker_image.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If our goal is to continue using our workspace, we should keep it available. On the contrary, if we plan on no longer using it and its associated resources, we can delete it.\n",
    "<br><br>\n",
    "Note: Deleting the workspace will delete all the experiments, outputs, models, Docker images, deployments, etc. that we created in that workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ws.delete(delete_dependent_resources=True)\n",
    "# This deletes our workspace, the container registry, the account storage, Application Insights and the key vault"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, Azure Container Instances tend to be used to develop and test deployments. They are typically configured with CPUs, which usually suffice when the number of requests per second is not too high. When working with several instances, we can configure them further by specifically allocating CPU resources to each of them.\n",
    "\n",
    "For production requirements, i.e. when > 100 requests per second are expected, we recommend deploying models to Azure Kubernetes Service (AKS). It is a convenient infrastructure as it manages hosted Kubernetes environments, and makes it easy to deploy and manage containerized applications without container orchestration expertise. It also supports deployments with CPU clusters and deployments with GPU clusters.For more examples on deployment follow [MachineLearningNotebooks](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/deployment) github repository.\n",
    "\n",
    "\n",
    "## Next Steps\n",
    "\n",
    "Check out [AutoML with Pipelines Deployment AKS](automl_with_pipelines_deployment_aks.ipynb) to see how to construct a AzureML Pipeline with an embedding step (using Google Universal Sentence Encoder model) and an AutoMLStep, increasing our Pearson correlation score. Also, this notebooks demonstrates deployment using AKS versus ACI."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
